Import necessary packages and libraries¶
In [1]:
pip install -U modin -q
In [2]:
pip install klib -q
In [3]:
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import klib
import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline
import missingno as msno
import warnings
warnings.filterwarnings('ignore')
from scipy import stats
from scipy.stats import anderson
from sklearnex import patch_sklearn
patch_sklearn()
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from timeit import default_timer as timer
import the dataset¶
The dataset contains almost 6 million records.Let's read the dataset using pandas library and see how long ita takes to read the dataset.¶
In [4]:
%time data=pd.read_csv("/kaggle/input/intel-oneapi-predict-the-quality-of-freshwater/dataset.csv")
Let's accelerate pandas using the intel distribution of modin and see the duration time.¶
In [5]:
import modin.pandas as pd
In [6]:
%time data=pd.read_csv("/kaggle/input/intel-oneapi-predict-the-quality-of-freshwater/dataset.csv")
The above information explain that the intel modin 80% faster than the pandas.¶
Let's see the structure of train dataset¶
In [7]:
%time data.info()
The dataset contains 5956842 rows and 24 columns.¶
Let's see column wise missing value count.¶
In [8]:
data.isnull().sum()
Out[8]:
Let's explore the column wise null values by bar chart.¶
In [9]:
msno.bar(data.drop(columns=['Index','Target']))
Out[9]:
Let's explore the null values correlation using heatmap.Null correlation explains how strong the non null values and null values strongly afftect the another columns.¶
In [10]:
msno.heatmap(data.drop(columns=['Index','Target']))
Out[10]:
The abov heatmap explains that thers is null value correlation between the columns.¶
Let's explore the null values using the dendrogram.The dendrogram reveal deeper null value trends.¶
In [11]:
msno.dendrogram(data.drop(columns=['Index','Target']))
Out[11]:
The above dendrogram explains that there are no columns grouped at level zero and the dataset columns null values are less likely correlated between th columns.¶
Let's see some samples from dataset.¶
In [12]:
data.head()
Out[12]:
Let's explore the Target column distribution.¶
In [13]:
sns.countplot(x=data['Target'].values)
Out[13]:
In [14]:
data['Target'].value_counts()
Out[14]:
The above plot explains that there is an imbalance between the classes.¶
Let's see the distribution of water's pH value.¶
In [15]:
def box_plot(df,col,rot=None):
_=plt.figure(figsize=(8,6))
_=sns.boxplot(y=df[col])
_=plt.title(col.capitalize()+" Distribution",fontsize=25)
_=plt.ylabel(col,fontsize=20,rotation=rot)
_=plt.yticks(fontsize=14)
In [16]:
def side_by_side_plot(df,grp,valcol,rot=None,title=""):
clr="Paired"
fig,(ax1,ax2) = plt.subplots(1,2,figsize=(18,8))
fig.tight_layout()
sns.kdeplot(x=df[valcol].values, hue=df[grp].values,ax=ax1,palette=clr)
ax1.set_title(grp.capitalize()+" Wise "+title+" Distribution",size=15)
ax1.set_xlabel(valcol,fontsize=20)
sns.boxplot(x=df[grp].values,y=df[valcol].values,ax=ax2,palette=clr)
ax2.set_title(grp.capitalize()+" Wise "+title+" Distribution",size=15)
ax2.set_xlabel(grp,fontsize=20)
ax2.tick_params(rotation=rot)
In [17]:
klib.dist_plot(data['pH']);
In [18]:
box_plot(data,'pH',rot=90)
In [19]:
data['pH'].describe()
Out[19]:
Let's see from what are the places the water samples are collected.¶
In [20]:
sns.countplot(y=data['Source'].values);
In [21]:
data['Source'].value_counts()
Out[21]:
The above plot explains that almost same amount of water samples are collected from different water sources.¶
Let's see are there any differences in pH value of water colected from different sources.¶
In [22]:
side_by_side_plot(data,'Source','pH',title="Water's Ph Value");
In [23]:
data.groupby('Color')['pH'].describe()
Out[23]:
The above plot and summary indicate that there are no significant differences between the source of the water and its pH measures.¶
Let's see the different water colors.¶
In [24]:
sns.countplot(y=data['Color'].values,order=data['Color'].value_counts().index);
In [25]:
data['Color'].value_counts()
Out[25]:
The above chart indicates that most of the water samples from various sources are either colorless or near colorless.¶
Let's see are there any differences in pH value of water and the color of water.¶
In [26]:
side_by_side_plot(data,'Color','pH',title="Water's Ph Value");
The above plot and summary indicate that there are no significant differences between the color of the water and its pH measures.¶
Let's see if is there any change in water pH value distribution by comparing with the source of water and the color of the water.¶
In [27]:
def water_source_color_measure(df,valcol):
fig=plt.subplots(figsize=(20, 20))
for i,cat in enumerate(['Lake', 'River', 'Ground', 'Spring', 'Stream', 'Aquifer','Reservoir', 'Well']):
_=plt.subplot(4,2,i+1)
_=sns.boxplot(x=df[df['Source']==cat]['Color'].values,y=df[df['Source']==cat][valcol].values)
_=plt.title(f"{cat} Water's {valcol} Value Distribution by Different Color",fontsize=15)
_=plt.xticks(fontsize=15)
_=plt.tight_layout()
plt.show()
In [28]:
water_source_color_measure(data,'pH');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on pH value.¶
Let's see the water's pH value distribution by the target class.¶
In [29]:
sns.kdeplot(x=data['pH'],hue=data['Target']);
In [30]:
data.groupby('Target')['pH'].describe()
Out[30]:
The above plot and summary indicate the pH value is not alone sufficient for determining whether the water is drinkable or not. Although, the pH value is an important indicator of water quality.¶
Let's see the distribution of water's Iron value(mscg/dl).¶
In [31]:
klib.dist_plot(data['Iron']);
In [32]:
box_plot(data,'Iron',rot=90);
In [33]:
data['Iron'].describe()
Out[33]:
Let's see if is there any change in water iron value distribution by comparing with the source of water and the color of the water.¶
In [34]:
water_source_color_measure(data,'Iron');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on iron value.¶
Let's see the water's iron value distribution by the target class.¶
In [35]:
sns.kdeplot(x=data['Iron'],hue=data['Target']);
In [36]:
data.groupby('Target')['Iron'].describe()
Out[36]:
The above plot and summary indicate the iron value is not alone sufficient for determining whether the water is drinkable or not. Although, the iron value is an important indicator of water quality.¶
Let's see the distribution of water's nitrate value(ppm).¶
In [37]:
klib.dist_plot(data['Nitrate']);
In [38]:
box_plot(data,'Nitrate',rot=90);
In [39]:
data['Nitrate'].describe()
Out[39]:
Let's see if is there any change in water nitrate value distribution by comparing with the source of water and the color of the water.¶
In [40]:
water_source_color_measure(data,'Nitrate');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on nitrate value.¶
Let's see the water's nitrate value distribution by the target class.¶
In [41]:
sns.kdeplot(x=data['Nitrate'],hue=data['Target']);
In [42]:
data.groupby('Target')['Nitrate'].describe()
Out[42]:
The above plot and summary indicate the nitrate value is not alone sufficient for determining whether the water is drinkable or not. Although, the nitrate value is an important indicator of water quality.¶
Let's see the distribution of water's chloride value(mEq/L).¶
In [43]:
klib.dist_plot(data['Chloride']);
In [44]:
box_plot(data,'Chloride',rot=90);
In [45]:
data['Chloride'].describe()
Out[45]:
Let's see if is there any change in water chloride value distribution by comparing with the source of water and the color of the water.¶
In [46]:
water_source_color_measure(data,'Chloride');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on Chloride value.¶
Let's see the water's chloride value distribution by the target class.¶
In [47]:
sns.kdeplot(x=data['Chloride'],hue=data['Target']);
In [48]:
data.groupby('Target')['Chloride'].describe()
Out[48]:
The above plot and summary indicate the chloride value is not alone sufficient for determining whether the water is drinkable or not. Although, the chloride value is an important indicator of water quality.¶
Let's see the distribution of water's lead value(µg/dL).¶
In [49]:
klib.dist_plot(data['Lead']);
In [50]:
box_plot(data,'Lead',rot=90);
In [51]:
data['Lead'].describe()
Out[51]:
Let's see if is there any change in water lead value distribution by comparing with the source of water and the color of the water¶
In [52]:
water_source_color_measure(data,'Lead');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on lead value.¶
Let's see the water's lead value distribution by the target class.¶
In [53]:
sns.kdeplot(x=data['Lead'],hue=data['Target']);
In [54]:
data.groupby('Target')['Lead'].describe()
Out[54]:
The above plot and summary indicate the lead value is not alone sufficient for determining whether the water is drinkable or not. Although, the lead value is an important indicator of water quality.¶
Let's see the distribution of water's zinc value(µg/mL).¶
In [55]:
klib.dist_plot(data['Zinc']);
In [56]:
box_plot(data,'Zinc',rot=90);
In [57]:
data['Zinc'].describe()
Out[57]:
Let's see if is there any change in water zinc value distribution by comparing with the source of water and the color of the water.¶
In [58]:
water_source_color_measure(data,'Zinc');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on zinc value.¶
Let's see the water's zinc value distribution by the target class.¶
In [59]:
sns.kdeplot(x=data['Zinc'],hue=data['Target']);
In [60]:
data.groupby('Target')['Zinc'].describe()
Out[60]:
The above plot and summary indicate the zinc value is not alone sufficient for determining whether the water is drinkable or not. Although, the zinc value is an important indicator of water quality.¶
Let's see the distribution of water's turbidity value(NTU).¶
In [61]:
klib.dist_plot(data['Turbidity']);
In [62]:
box_plot(data,'Turbidity',rot=90);
In [63]:
data['Turbidity'].describe()
Out[63]:
Let's see if is there any change in water turbidity value distribution by comparing with the source of water and the color of the water.¶
In [64]:
water_source_color_measure(data,'Turbidity');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on turbidity value.¶
Let's see the water's turbidity value distribution by the target class.¶
In [65]:
sns.kdeplot(x=data['Turbidity'],hue=data['Target']);
In [66]:
data.groupby('Target')['Turbidity'].describe()
Out[66]:
The above plot and summary indicate the turbidity value is not alone sufficient for determining whether the water is drinkable or not. Although, the turbidity value is an important indicator of water quality.¶
Let's see the distribution of water's fluoride value(mg/L).¶
In [67]:
klib.dist_plot(data['Fluoride']);
In [68]:
box_plot(data,'Fluoride',rot=90);
In [69]:
data['Fluoride'].describe()
Out[69]:
Let's see if is there any change in water fluoride value distribution by comparing with the source of water and the color of the water.¶
In [70]:
water_source_color_measure(data,'Fluoride');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on fluoride value.¶
Let's see the water's fluoride value distribution by the target class.¶
In [71]:
sns.kdeplot(x=data['Fluoride'],hue=data['Target']);
In [72]:
data.groupby('Target')['Fluoride'].describe()
Out[72]:
The above plot and summary indicate the fluoride value is not alone sufficient for determining whether the water is drinkable or not. Although, the Fluoride value is an important indicator of water quality.¶
Let's see the distribution of water's copper value(mg/L).¶
In [73]:
klib.dist_plot(data['Copper']);
In [74]:
box_plot(data,'Copper',rot=90);
In [75]:
data['Copper'].describe()
Out[75]:
Let's see if is there any change in water copper value distribution by comparing with the source of water and the color of the water.¶
In [76]:
water_source_color_measure(data,'Copper');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on copper value.¶
Let's see the water's copper value distribution by the target class.¶
In [77]:
sns.kdeplot(x=data['Copper'],hue=data['Target']);
In [78]:
data.groupby('Target')['Copper'].describe()
Out[78]:
The above plot and summary indicate the copper value is not alone sufficient for determining whether the water is drinkable or not. Although, the copper value is an important indicator of water quality.¶
Let's see the distribution of water's Odor value(ou/m3).¶
In [79]:
klib.dist_plot(data['Odor']);
In [80]:
box_plot(data,'Odor',rot=90);
In [81]:
data['Odor'].describe()
Out[81]:
Let's see if is there any change in water odor value distribution by comparing with the source of water and the color of the water.¶
In [82]:
water_source_color_measure(data,'Odor');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on odor value.¶
Let's see the water's odor value distribution by the target class.¶
In [83]:
sns.kdeplot(x=data['Odor'],hue=data['Target']);
In [84]:
data.groupby('Target')['Odor'].describe()
Out[84]:
The above plot and summary indicate the odor value is not a sufficient indicator for determining whether the water is drinkable or not.¶
Let's see the distribution of water's sulfate value(mg/l).¶
In [85]:
klib.dist_plot(data['Sulfate']);
In [86]:
box_plot(data,'Sulfate',rot=90);
In [87]:
data['Sulfate'].describe()
Out[87]:
Let's see if is there any change in water sulfate value distribution by comparing with the source of water and the color of the water.¶
In [88]:
water_source_color_measure(data,'Sulfate');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on sulfate value.¶
Let's see the water's sulfate value distribution by the target class.¶
In [89]:
sns.kdeplot(x=data['Sulfate'],hue=data['Target']);
In [90]:
data.groupby('Target')['Sulfate'].describe()
Out[90]:
The above plot and summary indicate the sulfate value is not a sufficient factor for determining whether the water is drinkable or not.¶
Let's see the distribution of water's conductivity value(µmhos/cm).¶
In [91]:
klib.dist_plot(data['Conductivity']);
In [92]:
box_plot(data,'Conductivity',rot=90);
In [93]:
data['Conductivity'].describe()
Out[93]:
Let's see if is there any change in water conductivity value distribution by comparing with the source of water and the color of the water.¶
In [94]:
water_source_color_measure(data,'Conductivity');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on conductivity value.¶
Let's see the water's conductivity value distribution by the target class.¶
In [95]:
sns.kdeplot(x=data['Conductivity'],hue=data['Target']);
In [96]:
data.groupby('Target')['Conductivity'].describe()
Out[96]:
The above plot and summary indicate the conductivity value is not alone sufficient for determining whether the water is drinkable or not. Although, the conductivity value is an important indicator of water quality.¶
Let's see the distribution of water's chlorine value(mg/L).¶
In [97]:
klib.dist_plot(data['Chlorine']);
In [98]:
box_plot(data,'Chlorine',rot=90);
In [99]:
data['Chlorine'].describe()
Out[99]:
Let's see if is there any change in water chlorine value distribution by comparing with the source of water and the color of the water.¶
In [100]:
water_source_color_measure(data,'Chlorine');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on Chlorine value.¶
Let's see the water's chlorine value distribution by the target class.¶
In [101]:
sns.kdeplot(x=data['Chlorine'],hue=data['Target']);
In [102]:
data.groupby('Target')['Chlorine'].describe()
Out[102]:
The above plot and summary indicate the chlorine value is not alone sufficient for determining whether the water is drinkable or not. Although, the chlorine value is an important indicator of water quality.¶
Let's see the distribution of water's manganese value(mg/L).¶
In [103]:
klib.dist_plot(data['Manganese']);
In [104]:
box_plot(data,'Manganese',rot=90);
In [105]:
data['Manganese'].describe()
Out[105]:
Let's see if is there any change in water manganese value distribution by comparing with the source of water and the color of the water.¶
In [106]:
water_source_color_measure(data,'Manganese');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on manganese value.¶
Let's see the water's manganese value distribution by the target class.¶
In [107]:
sns.kdeplot(x=data['Manganese'],hue=data['Target']);
In [108]:
data.groupby('Target')['Manganese'].describe()
Out[108]:
The above plot and summary indicate the manganese value is not alone sufficient for determining whether the water is drinkable or not. Although, the manganese value is an important indicator of water quality.¶
Let's see the distribution of water's total dissolved solids value(milligrams per liter).¶
In [109]:
klib.dist_plot(data['Total Dissolved Solids']);
In [110]:
box_plot(data,'Total Dissolved Solids',rot=90);
In [111]:
data['Total Dissolved Solids'].describe()
Out[111]:
Let's see if is there any change in water total dissolved solids value distribution by comparing with the source of water and the color of the water.¶
In [112]:
water_source_color_measure(data,'Total Dissolved Solids');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on total dissolved solids value.¶
Let's see the water's total dissolved solids value value distribution by the target class.¶
In [113]:
sns.kdeplot(x=data['Total Dissolved Solids'],hue=data['Target']);
In [114]:
data.groupby('Target')['Total Dissolved Solids'].describe()
Out[114]:
The above plot and summary indicate the total dissolved solids value is not alone sufficient for determining whether the water is drinkable or not. Although, the total dissolved solids value is not used as a sole indicator to determine water quality.Instead, a combination of TDS value and other measures like pH value will be considered.¶
Let's see the distribution of water's temperature value(c).¶
In [115]:
klib.dist_plot(data['Water Temperature']);
In [116]:
box_plot(data,'Water Temperature',rot=90);
In [117]:
data['Water Temperature'].describe()
Out[117]:
Let's see if is there any change in water temperature value distribution by comparing with the source of water and the color of the water.¶
In [118]:
water_source_color_measure(data,'Water Temperature');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on temperature value.¶
Let's see the water's temperature value distribution by the target class.¶
In [119]:
sns.kdeplot(x=data['Water Temperature'],hue=data['Target']);
In [120]:
data.groupby('Target')['Water Temperature'].describe()
Out[120]:
The above plot and summary indicate the temperature value is not determining whether the water is drinkable or not.¶
Let's see the distribution of water's air temperature value(c).¶
In [121]:
klib.dist_plot(data['Air Temperature']);
In [122]:
box_plot(data,'Air Temperature',rot=90);
In [123]:
data['Air Temperature'].describe()
Out[123]:
The above histogram plot explain that the air temperature column is normally distributed.¶
Water's air temperature value ranges from -33.8 to 152.124.¶
The average air temperature value of water is 60.00324.¶
The boxplot explains that there are outliers above the third quartile brlow the first quartile.¶
Let's see if is there any change in water's air temperature value distribution by comparing with the source of water and the color of the water.¶
In [124]:
water_source_color_measure(data,'Air Temperature');
The above plot explains that there is no significant interaction effect between the color of the water and the water source on air temperature value.¶
Let's see the water's air temperature value distribution by the target class.¶
In [125]:
sns.kdeplot(x=data['Air Temperature'],hue=data['Target']);
In [126]:
data.groupby('Target')['Air Temperature'].describe()
Out[126]:
The above plot and summary indicate the air temperature value is not determining whether the water is drinkable or not.¶
In [127]:
def w_test(df,targetcol,numcol):
t_stat, p_val = stats.ranksums(df[df[targetcol] == 1][[numcol,targetcol]].dropna()[numcol],
df[df[targetcol] == 0][[numcol,targetcol]].dropna()[numcol])
return [t_stat ,p_val]
Let's find the statistical significance between the independent and dependent column.¶
In [128]:
t_stat=[]
p_val=[]
col_list=['pH', 'Iron', 'Nitrate', 'Chloride', 'Lead', 'Zinc',
'Turbidity', 'Fluoride', 'Copper', 'Odor', 'Sulfate', 'Conductivity',
'Chlorine', 'Manganese', 'Total Dissolved Solids',
'Water Temperature', 'Air Temperature']
for col in col_list:
#res=[]
#t_stat, p_val = w_test(data,"Target",col)
result=w_test(data,"Target",col)
t_stat.append(result[0])
p_val.append(result[1])
In [129]:
stat_test=pd.DataFrame({'column_name':col_list,'t_tstat':t_stat,'p_value':p_val})
stat_test['result']=stat_test['p_value'].apply(lambda x:"significant" if x<0.05 else "not_significant" )
In [130]:
stat_test
Out[130]:
The above summary explains that there is no significant difference between the following columns and the target column.¶
- Conductivity
- Water Temperature
- Air Temperature